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■  Abstract 


"'  '^System?  with  wide  bandwidth  noise  inputs  are  a  common  occurrence  in 
stochastic  control  and  communication  theory  and  elsewhere;  e.g.,  tracking 
or  syTichronication  systems  such  as  phase  locked  loops  (PLL) .  One  is  often 
interested  in  calculating  such  quantities  as  the  probability  of  escape 
from  a  desired  'error'  set,  in  some  time  interval,  or  the  mean  time  for 
such  escape.  Diffusion  approximations  (the  system  obtained  in  the  limit 
bandwidth  •*  ®)  are  often  used  for  this,  being  easier  to  analyze.  When  the 
noise  effects  in  the  physical  system  are  small,  one  is  tempted  to  do  an 
asymptotic  analysis  (noise  intensity  -*•  0)  on  the  diffusion  approximation, 
and  use  this  for  the  desired  estimates  on  the  original  system.  Such  a 
procedure  does  not  work  in  general;  the  double  limit  bandwidth  -*•  ®, 
intensity  0  is  not  always  justified.  Under  quite  broad  conditions  on 
the  noise  processes,  it  is  justified  for  the  systems  studied  here.  We 
study  a  particular  form  of  the  PLL  owing  to  the  great  practical  importance 
of  the  system  and  because  it  provides  a  useful  vehicle  for  understanding 
the  extent  of  validity  of  the  asymptotic  methods  for  such  systems.  The 
basic  analytical  techniques  are  from  the  theory  of  large  deviations.  One 
seeks  information  on  the  escape  probabilities,  mean  times,  and  on  the  most 
likely  exit  paths  and  exit  locations.  Also,  we  seek  information  on  the 
interactions  between  the  signals  to  be  tracked  and  the  noise  which  are  most 
likely  to  lead  to  exit.  The  large  deviations  technique  is  eminently 
suited  to  this  job. 

Simulations  are  taken  in  order  to  understand  the  range  of  validity  of 
the  asymptotic  method.  Agreement  between  the  predictions  and  sample  esti¬ 
mates  is  good  over  noise  intensity  levels  which  seem  to  be  ever  larger  than 
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those  typically  occurring  in  practice.  Since  the  events  whose  probability 
is  of  interest  have  small  probability.^'atTlmportance  sampling  scheme  is 
used  to  'quicken'  the  simulations.  The  scheme  works  well.  The  required 
neasure  transformations  are  suggested  by  the  theory  of  large  deviations, 
and  are  obtained  by  solving  the  associated  variational  problem.  The 
technique  seems  to  be  very  appropriate  as  both  an  analysis  and  design 
tool  for  such  systems. 


■  Key  Words:  large  deviations,  approximations  of  systems  with  wide 
1'^'  bandwidth  noise,  importance  sampling,  quick  simulation,  small  noise 

■  diffusion  models,  asymptotic  methods,  phase  locked  loops,  synchronization 
systems . 


WLV’A  tL^JV.  W  W  V  H."  ii*V  ■?• 


STOCHASTIC  SYSTEMS  KITH  SmLL  NOISE,  ANALYSIS 
AND  SIMULATION;  A  PHASE  LOCKED  LOOP  EXAMPLE 


by 

P.  Dupuis  and  H.  Kushner 


August  1985 


LCDS  #85-20 


AIW  roiwn  orrz  rn  o»  sciwtwxc  CTSitJrn  ( 
mrics  Of  nui»3ai?TiL  ^  ^  ^  ^ 

This  t*<: hot •>-.i  i4«trAl»{>-ia. 

DlBtrlKutlon  iJf  unll»lt«d. 


STOCHASTIC  SYSTEMS  KITH  SMALL  NOISE,  ANALYSIS 
AND  SIMULATION;  A  PHASE  LOCKED  LOOP  EXAMPLE 


P.  Dupuis* 

H.  Kushner** 


Lefschetz  Center  for  Dynamical  Systems 
Division  of  Applied  Mathematics 
Brown  University 
Providence,  Rhode  Island  02912 


August  1985 


This  reserach  was  supported  in  part  by  the  Army  Research  Office  under 
'Contract  No.  DAAG  29-84-K-0082,  the  Office  of  Naval  Research  under 
Contract  No.  N00014'83-K-0542,  and  the  National  Science  Foundation  under 
Grant  No.  DMS  85-11470. 


This  research  was  supported  in  part  by  the  National  Science  Foundation 
under  Grant  No.  ECS  82-11476  and  the  Air  Force  Office  of  Scientific 
Research  under  Contract  No.  AF-AFOSR  81-0116. 


Abstract 


Systems  with  wide  bandwidth  noise  inputs  are  a  common  occurrence  in 
stochastic  control  and  communication  theory  and  elsewhere;  e.g.,  tracking 
or  synchronization  systems  such  as  phase  locked  loops  (PLL) .  One  is  often 
interested  in  calculating  such  quantities  as  the  probability  of  escape 
from  a  desired  ’error’  set,  in  some  time  interval,  or  the  mean  time  for 
such  escape.  Diffusion  approximations  (the  system  obtained  in  the  limit 
bandwidth  ->■  are  often  used  for  this,  being  easier  to  analyze.  When  the 
noise  effects  in  the  physical  system  are  small,  one  is  tempted  to  do  an 
asymptotic  analysis  (noise  intensity  0)  on  the  diffusion  approximation, 
and  use  this  for  the  desired  estimates  on  the  original  system.  Such  a 


procedure  does  not  work  in  general:  the  double  limit  bandwidth  ■* 
intensity  ->-0  is  not  always  justified.  Under  quite  broad  conditions  on 
the  noise  processes,  it  is  justified  for  the  systems  studied  here.  We 
study  a  particular  form  of  the  PLL  owing  to  the  great  practical  importance 
of  the  system  and  because  it  provides  a  useful  vehicle  for  understanding 
the  extent  of  validity  of  the  asyinptotic  methods  for  such  systems.  The 
basic  analytical  techniques  are  from  the  theory  of  large  deviations.  One 
seeks  information  on  the  escape  probabilities,  mean  times,  and  on  the  most 
likely  exit  paths  and  exit  locations.  Also,  we  seek  information  on  the 
interactions  between  the  signals  to  be  tracked  and  the  noise  which  are  most 
likely  to  lead  to  exit.  The  large  deviations  technique  is  eminently 
suited  to  this  job. 

Simulations  are  taken  in  order  to  understand  the  range  of  validity  of 
the  asymptotic  method.  Agreement  between  the  predictions  and  sample  esti¬ 
mates  is  good  over  noise  intensity  levels  which  seem  to  be  ever  larger  than 
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those  typically  occurring  in  practice.  Since  the  events  whose  probability 
is  of  interest  have  small  probability,  an  importance  sampling  scheme  is 
used  to  'quicken'  the  simulations.  The  scheme  works  well.  The  requi'^ed 
measure  transformations  are  suggested  by  the  theory  of  large  deviatic 
and  are  obtained  by  solving  the  associated  variational  problem.  The 
technique  seems  to  be  very  appropriate  as  both  an  analysis  and  design 
tool  for  such  systems. 


Key  Words:  large  deviations,  approximations  of  systems  with  wide 
bandwidth  noise,  importance  sampling,  quick  simulation,  small  noise 
diffusion  models,  asymptotic  methods,  phase  locked  loops,  synchronization 


systems . 


I.  Introduction 


Let  C^  (')  denote  a  wide  bandwidth  (BW)  process  (BW  «  as  X  0)  , 
{5^,  0i:n<“>}  a  sequence  of  correlated  random  variables  for  each  X  >  0, 
and  w(*)  a  standard  Wiener  process.  Systems  of  the  types  (1.1)  to  (1.3) 
are  frequently  used  models  in  stochastic  control  and  communication  theory 
and  elsewhere 


(1.1)  =  F^(x\c^) 

(1.2)  +  XF,(X\c^) 

^  n+1  n  X^  n  n 

(1.3)  dx  =  f(x)dt  +  o(x)dw 


The  function  F^  in  (1.1)  is  such  that  for  small  X,  the  process  x^(0 
is  close  (say,  in  the  sense  of  weak  convergence)  to  the  diffusion  x(*)- 
Similarly  for  the  F^  in  (1.2). 

Typically,  (1.3)  is  used  as  an  approximation  to  the  'physical'  models 
(1.1)  or  (1.2).  Signal  tracking  and  synchronization  systems  (with  noise 
corrupted  observations)  represent  one  very  important  class  of  applications 
in  communication  theory;  see,  for  example,  the  enormous  literature  on  phase 
locked  loops  (PLL) ,  [1]  to  [4].  For  such  systems,  one  or  more  of  the  com¬ 
ponents  of  the  state  x  in  (1.1)  might  represent  the  error  in  tracking  a 
signal,  and  the  others  might  represent  internal  states  of  the  system  (the 
filter,  etc.).  TN'pically,  one  is  interested  in  estimates  of  quantities 
such  as 

(1.4)  P^{x^(t)  i  G,  some  t  <  T) 

E  T^,  where  =  inf  {t:  x^(t)  4  G)  , 


(1.5) 


for  suitable  sets  G  and  times  T.  Estimates  of  such  quantities  are 
generally  quite  hard  to  get  for  any  of  the  models  (1.1)  to  (1.3),  but 
certainly  much  harder  (if  not  impossible)  for  the  wide  BK  driven  system 
(1.1)  (or  (1.2)),  than  for  (1.3).  Owing  to  this,  one  is  often  forced  to 
use  some  sort  of  as>'mptotic  method. 

Since  (1.3)  is  much  easier  to  work  with  than  (1.1)  or  (1.2)  one  is 
strongly  tempted  to  find  a  diffusion  approximation  to  x^(*)  or  to  {X^}. 
If  the  noise  effects  are  not  'small,'  then  the  diffusion  approximation 
method  can  give  good  results  [19],  [20].  But  even  for  this  case,  it 
might  be  hard  to  solve  for  the  analogs  of  (1.4),  (1.5): 

(1.4')  Pjj{x(t)  i  G,  some  t  ?  T) 

(1.5')  =  inf  (t:  x(t)  4  Gl  . 

Often,  one  estimates  quantities  such  as  (1,4'),  (1.5')  by  assuming 
that  the  noise  effects  are  small;  e.g.,  by  using  the  model  (1.6),  and 
estimating  (1.7),  (1.8)  via  some  other  asymptotic  method. 

(1.6)  dx^  =  f(x^)dt  +  eo(x'')dw' 

(1.7)  P{x^(t)4G,  some  t  j  T) 

(1.8)  E^Tg  ,  =  inf  (t:  x^(t)  4  G}  . 

We  have  mentioned  two  very  different  t>’pes  of  approximations.  First, 
letting  BW  -*■  <=  to  approximate  (1.1)  by  (1.3),  then  letting  the  noise 
intensity  go  to  zero  (replacing  o  by  eo)  to  get  an  as>'mptotic  estimate 
for  (1.7),  (1.8).  This  procedure  is  rather  risky  in  that  the  estimates  of 
(1.7),  (1.8)  might  be  very  poor  estimates  of  (1.4),  (1.5),  even  when  the 


3 


'intensity'  of  the  noise  effects  in  (1.1)  is  small.  See  [5]  for  some 
examples.  Basically,  when  c  or  the  noise  effects  in  (1.1)  are  small,  the 
events  of  interests  are  rare  (e.g.,  the  event  in  (1.4)  or  (1.7))  or  the 
times  (1.5),  (1.8)  very  large.  The  escape  phenomenon  depends  on  rarely 
occurring  large  bursts  of  noise,  and  the  usual  'central  limit  theorem  like' 
arguments  which  are  used  to  model  (1.1)  by  (1.3)  do  not  model  these  well. 

In  order  to  use  an  asymptotic  expansion  based  on  (1.6)  for  estimates 
of  functionals  of  (1.1),  one  must  show  that  it  yields  the  same  results  as 
one  would  get  from  (1.1)  if  the  BK  went  to  and  the  noise  intensity 
went  to  zero  simultaneously  (with  an  analogous  result  for  (1.2)),  not 
first  letting  BW  ->■  «■,  then  intensity  -►  0.  For  a  large  class  of  noise 
processes,  such  results  are  in  [6].  These  results  seem  to  be  crucial  if 
one  is  to  employ  as.vmptotic  methods  for  (1.6)  to  get  properties  of  (1.1). 

One  interesting  asymptotic  method  based  on  (often  formal)  expansions 
and  boundary  layer  matchings  is  discussed  in  [7]-[8].  A  rigorous  approach 
to  such  a  scheme  involves  strong  regularity  and  ellipticity  conditions  on 
the  differential  generator  of  (1.3).  An  alternative  approach  is  via  the 
theory  of  large  deviations,  where  the  estimates  (1.4),  (1.5)  or  (1.7), 
(1.8),  involve  the  solution  to  a  variational  problem  [6],  [9],  [10],  [11]. 

A  related  small  noise  expansion  is  in  [12]. 

Once  the  small  noise  (small  e)  expansion  is  obtained,  one  must  ask 
how  good  it  is  for  values  of  e  which  one  might  expect  in  applications. 

To  get  such  information,  there  seems  (at  the  moment)  little  alternative  to 
simulation.  The  questions  of  model  approximation  and  validation  by  simula¬ 
tion  will  be  addressed  here  for  one  particular  PLL,  CXjr  PLL  model  is 
chosen  both  because  it  is  important  in  applications  and  because  it  provides 
a  good  illustration  of  the  general  method. 
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In  Section  2,  we  formulate  the  particular  PLL  problem  of  interest 
here.  The  diffusion  approximation  form  of  this  system  is  not  of  the  form 
(1.3)  unless  various  ('double  frequency')  terms  are  dropped,  the  system 
reduced  to  'baseband'  form,  and  the  wide  bandwidth  'small  intensity' 
observation  noise  replaced  by  small  intensity  white  Gaussian  noise.  This 
procedure  is,  in  fact,  valid  for  our  case,  under  quite  broad  conditions. 
Reference  [13]  deals  with  a  related  problem,  and  below  we  comment  further 
on  the  relations  between  that  work  and  ours.  The  basic  scheme  of  the 
theory  of  large  deviations  and  the  simplifications  of  our  model  is  dis¬ 
cussed  in  Section  3. 

Generally,  with  small  noise  effects,  the  'escape'  probabilities  of 
the  forms  (1.4)  (or  (1.7))  are  very  small;  conversely  (1.5)  (or  (1.8)) 
would  be  quite  large.  In  order  to  get  good  simulation  results  with  a 
reasonable  amount  of  computational  effort,  a  form  of  importance  sampling 
is  used.  The  idea  is  to  change  the  measure  on  the  original  probability 
space  in  a  way  which  increases  the  probability  of  the  'rare'  escape  event 
on  the  time  interval  [0,T].  Then,  we  get  the  required  estimate  by  an 
inverse  transformation.  A  natural  way  in  which  to  do  this  is  suggested  by 
the  mathematics  of  the  theory  of  large  deviations,  and  (as  will  be  seen) 
works  very  well.  The  idea  was  first  formally  used  in  [14],  and  is  dis¬ 
cussed  in  Section  4.  Simulation  results  for  the  PLL  (with  a  constant  phase 
process)  appear  in  Section  5.  It  is  remarkable  that  the  estimates  given  by 
the  asymptotic  theory  are  quite  good  even  for  rather  large  values  of  e. 

Let  0(*)  and  6^(*)  (for  model  (1.6),  or  6^(*)  if  the  model  is 
(1.1))  denote  the  phase  to  be  tracked  and  its  estimate,  as  given  by  the 
PLL.  The  error  0(*)  -  0  (•)  (or  0(*)  -  0  (•))  will  be  a  component  of 
the  state  x,  and  we  use  sets  G  of  roughly  the  form  G  =  G_  =  {x:  |6-6|  <  "•) 
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This  set  is  standard  in  the  study  of  the  PLL,  since  when  the  error  reaches 
n-  the  system  can  'lose  track.'  An  advantage  of  the  large  deviations 
approach  is  that  one  can  (rigorously)  get  information  on  the  most  likely 
locations  of  points  of  escape  from  G,  on  the  paths  in  whose  (small) 
neighborhoods  escape  is  most  likely,  and  on  the  most  likely  magnitudes  of 
the  noise  effects  when  escape  occurs.  When  the  signal  is  time  varying,  the 
scheme  gives  the  most  likely  interactions  between  the  signal  and  noise 
which  lead  to  escape. 

In  Sections  6  and  7,  we  discuss  the  results  when  the  phase  varies  as 
it  would  in  a  digital  pulse  phase  modulation  system.  The  model  and  the 
simulations  are  discussed  and  one  can  clearly  see  the  interactions  between 
the  anticipated  changes  in  the  signal  and  the  noise  processes  which  are 
most  likely  to  lead  to  escape. 

An  interesting  study  of  a  related  problem  is  in  [13].  The  PLL  in 
[13]  is  somewhat  more  complicated  than  ours.  They  start  with  a  'baseband' 
system  of  4th  order  and  reduce  it  to  a  2nd  order  system.  The  input  signal 
has  wide  BK  of  0(K)  ,  where  k  is  also  a  system  gain,  and  the  observation 
noise  intensity  (/e  is  used  there  rather  than  e)  is  small.  Since  the 
system  is  not  of  the  form  (1.6),  a  scale  change  in  both  time  and  state  is 
made  (in  an  c,  k-dependent  way)  in  order  to  get  a  diffusion  model  whose 

e 

driving  Wiener  process  is  multiplied  by  a  small  parameter.  Let  L  denote  the 
differential  generator  of  the  rescaled  system--which  we  can  suppose  has  the 
form  (1.6).  Let  V^(x)  =  Then,  formally,  L^'V^  (x)  +  1  =  0,  x  e  G, 

V^(x)  =  0,  X  £  3G.  Write  V^(x)  =  exp  [^  +  C(x)  +  higher  order  terms]. 

Via  a  formal  expansion  and  boundary  layer  matching  method,  an  interesting 
(but  heuristic)  approximation  for  C(x)  and  H  are  obtained.  Their  simu¬ 
lations  were  in  good  agreement  with  the  theoretical  predictions  over  the 
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range  of  e  used.  Although  the  systems  are  hard  to  compare,  it  seems  that 
our  range  of  e  (or  at  least  our  range  of  noise  effects)  is  somewhat 
larger,  if  one  uses  (e.g.)  estimates  of  (1.7),  (1.8)  as  a  measure  of  noise 
effects.  The  results  are  hard  to  compare  directly  with  ours,  since  the 
system  ty'pes  are  quite  different,  and  because  of  the  way  e  and  k  are 
buried  in  the  time  and  state  scaling  in  [13] . 

Our  method  would  not  get  C(x),  but  the  importance  of  this  term, 
in  general,  is  not  clear.  If  e  is  large,  then  the  (small  e)  approxima¬ 
tion  to  this  term  might  be  unreliable.  If  e  is  small,  the  overwhelming 

2 

effects  are  due  to  exp  H/e 


I 


a 


2.  The  PLL:  The  Model  and  Approximation 


The  basic  physical  system  model  is  in  Fig.  2.1.  The  VCO  (voltage 

controlled  oscillator)  is  a  device  whose  output  deviates  from  a  reference 

frequency  according  to  the  input  voltage.  In  order  to  get  an  asymptotic 

analysis  from  a  reasonably  practical  perspective  we  proceed  as  follows, 

using  the  fact  that  for  many  practical  systems,  the  carrier  frequency  is 

high  and  the  BW  narrow  relative  to  the  carrier  but  high  in  an  absolute 

sense.  We  use  a  (high)  carrier  frequency  and  an  observation  noise 

process  whose  BW  is  of  the  order  of  l/y,  but  is  small  relative  to  . 

Let  n  be  such  that  q  /v  -*■  0  as  y  -*■  0,  and  let  =  w^/n'''^  for  some 

y  y  0 


Let  C^(’).  i  =  i-2,  be  mutually  independent  zero  mean  second  order 
stationary  processes.  The  BW  of  cT(*)  will  be  of  0(1/y).  Let  p  =  (e,Y) 
Following  a  common  practice  in  communication  theory,  we  model  the  observa¬ 
tion  noise  in  the  'passband'  form  n‘^(t)  =  Eu''^(t),  where 


(2.1) 


u''^(T)  =  Cj(t)  sin  u^t  +  cos  u)^t 


The  e  indexes  the  intensity  of  the  noise  and  y  the  BW.  The  following 

•y 

two  models  for  can  be  used  and  cover  many  cases  in  practice.  For 

case  1,  let  cT(t)  =  5^(t/Y)/*^  ,  where  Cj(’)  is  a  component  of  a 

stationary  Gauss-Markov  process  with  an  integrable  correlation  function, 
t 

Write  wT(t)  =  /  C^(s)  ds.  Then  the  pair  (w^(*) .W2(’))  converges  weakly 

to  a  pair  of  mutually  independent  Wiener  processes,  each  of  which  has 
00 

covariance  /  EC(s)C(0)ds  =  .  This  is  one  standard  way  of  modelling 

•  00 

wide  BW  noise.  We  prefer  to  separate  the  parameters  for  intensity  and  BW , 
although  they  can  be  combined.  Similarly  for  the  case  below. 


FIGURE  2.1  A  PHASE  LOCKED  LOOP 
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Much  physical  noise  has  an  'impulsive'  origin.  The  'raw'  impulses  are 
either  filtered  by  a  circuit  or  system,  or  else  the  impulses  are  actually 
pulses  whose  intensity  decreases  rapidly.  To  model  this,  we  use  the 
second  case  where  cT(t)  =  ?T(t/y)/Y.  where  ^(*)  is  a  filtered  impulsive 
noise  process,  the  jump  rate  and  moments  depending  on  y  (increasing  and 
decreasing,  respectively,  as  y  -♦  0) .  The  specific  model  is  Case  II  of  [6]. 
The  pair  (w^(  •)  .*''2 ( *))  again  converges  (under  appropriate  conditions  on 
the  jump  rate  and  moments)  to  a  pair  of  independent  Wiener  processes  with 
covariance  .  See  [6]  for  more  detail. 

The  input  to  our  PLL  (signal  plus  noise)  is 

u^(t)  =  Aq  sin  (oj^t  +  6)  +  eu^(t) 

and  the  system  equations  are  (see  Fig.  1) 

(2.2)  v^  =  Av*^  +  B  cos  (u^t  +  (t) 

=  Hv^  , 

where  A  is  a  stable  matrix.  Owing  to  the  complicated  form  of  the 
expression  (2.3),  (2.2)  is  not  suitable  for  analysis  or  computation. 

cos  (u^t  +  6^)u^ (t)  = 

A  ^  A 

sin  (e-6*^)  +  —  sin  (6+6^+2a)^t) 

(2.3) 

+  f  cos  sin  e^j 

+  y  [c|(t)  cos  (2uj^t  +  e^)  ♦  sin  (2a)^t  +  e^’)] 

If  we  were  justified  in  dropping  the  parts  of  (2.3)  containing  w  ,  then 
we  could  replace  (2.2)  bv  the  system 


W  ' 

•  ", 

r'vl' 


Vv 


(2.4) 


=  Av^  +  B|^  sin  (6-6^) 

+  I  [c](t)  cos  6^  -  C^(t)  sin  eP] 

e'^  =  Hv^  . 

If,  furthermore,  we  were  justified  in  replacing  the  (2.4) 

by  white  Gaussian  noise  of  intensity  o,  we  could  use  the  model  (2.5)  for 
the  'asymptotic'  calculations. 

A 

(2.5)  dv'^  =  Av^dt  +  B  ^  sin  (e-e^)dt 
+  B  y  odw 

6^  =  Hv^  ,  (6-6^)  =  -Hv^  if  e(t)  =  constant. 

It  turns  out  that  one  can,  in  fact,  make  all  of  these  approximations 
in  the  sense  that  the  asymptotic  (e-*-0,  y 0)  estimates  of  the  escape 
probabilities  and  times  will  be  close.  The  sense  of  'closeness'  is  dis¬ 
cussed  in  the  next  section,  after  we  introduce  some  concepts  from  the 
theory  of  large  deviations. 

With  arbitrary  WB  noise  models  ^j(*)  such  that  w^(*)  converges 
weakly  to  a  Wiener  process,  the  estimates  of  escape  probabilities  and  times 
for  (2.2)  or  (2.5)  might  be  very  different  for  small  e,^.  For  example, 
if  C|(*)  is  a  scaled  continuous  parameter  Markov  chain  or  if  the 
moments  of  the  impulses  in  case  2  increase  too  fast  [6].  One  must 
exercise  considerable  care  in  replacing  (2.2)  by  (2.5). 
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3 .  Large  Deviations  and  System  Approximations 


Refer  to  (1.5)  and  (1.6)  and  let  a(*)  and  £(•)  satisfy  a  uniform 

Lipschitz  and  a  linear  growth  condition  (i.e.,  |f(x)|  <  K(l+|x|)). 

Define  the  functionals  H(*,*)  and  L(*,*)  by  H(oi,x)  =  a'f (x)+a'o(x)a' (x)a/ 

and  L(S,x)  =  sup  [ 6 ' a-H(a ,x) ] .  In  our  PLL  model  the  driving  noise  is 
a 

degenerate.  To  prepare  for  this,  consider  the  special  case  where 
(x^.x.,)  =  x  and  OjOj  is  uniformly  positive  definite  and 

(3.1)  dx^  =  f^(x)dt  +  £aj(x)dw 
dx^  =  f^(x)dt  . 

Split  a  =  (a^.a.,),  etc.  Then 

H(a,x)  =  ajIfjCx)  +  a^f2(x)  +  a|Oj  (x)  o|  (x)Oj/2 
and 

(3.2)  L(e,x)  =  j  (6^-fj(x))’  (Oj(x)o’(x))'^(e^*f^(x)) 

if  ^2  = 

=  «  otherwise. 


Define  the  action  functional 

rT 


(3.3)  S^(T,(^)  = 


0 


l.(i(s)  ,<ti(s))ds  ,  for 


41(0)  =  x  ,  (!»(•)  absolutely  continuous 


00 


Otherwise . 
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Let  X  =  0  be  an  as>Tnptotical ly  stable  point  of  x  =  f(x)  with  G 
being  a  bounded  open  set  whose  closure  G  is  in  the  domain  of  attraction 
of  {0}.  Define 


S^(T)  =  i^f  S^(T,(^) 


(3.4) 


S  =  lim  S  (T)  , 

X  „  x"-  ’ 

T-w 


where  the  inf  is  over  all  (()(•)  which  escape  G  by  time  T.  Then, 
under  broad  conditions  on  G  [9,10] 


(3.5a)  lim  log  P^{x^(t)4G,  some  tjT)  =  -S^(T) 


(3.5b)  lim  jog  E  ,  x e  G  , 

If  (3.5)  does  not  hold  for  a  particular  set  G,  it  will  hold  for  a  small 
perturbation  of  G.  A  maximum  likelihood  interpretation  of  (3.5)  appears  in 
the  next  section. 

Now,  let  us  relate  the  above  facts  to  the  system  (2.5)  Let  e(t)  =  6, 
a  constant.  For  notational  convenience  absorb  the  Ap/2  into  B  and  the 
o/2  into  e,  and  rewrite  (2.5)  as 


(3.6)  dv^  =  Av^dt  +  B  sin  (e-6^)dt  +  Bedw 


(e-e^)  =  -Hv^  dt  . 

A  A 

Write  X  =  (v,0-6).  The  set  G  of  interest  here  is  G  =  G^  =  {x:  |6-0l  <  it)  fl  G2, 
where  G2  is  the  domain  of  attraction  of  the  stable  point  x  =  0.  The 
closure  G,  is  compact,  but  not  all  of  it  is  in  the  domain  of  attraction  of  (0), 
since  (0,±it)  is  also  a  singular  point.  Nevertheless,  owing  to  the  special  form, 
the  additional  problems  are  minor  and  (3.5)  holds. 


The  approximation  (2.5)  to  (2.2)  is  valid  [6]  in  the  sense  that,  for 
xeG  (p -*•  0  implies  e-*-0,  X ->•  0) 

(3.7a)  lim  log  {x*^(t)  ^  ,  some  t  <  T] 

=  lim  log  P^{x^(t)  4  Gj  ,  some  t  ^  T} 

(3.7b)  lim  log  E  =  lim  log  E 

p  X  U2  £  X  ^2 

The  proof  of  (3.7)  involves  details  from  the  theory  of  large 
deviations  [6] .  But  without  such  a  proof  one  could  not  justify  using  the 
simpler  system  (2.5)  for  computational  purposes.  The  values  of  (1,4), 
(1.5)  depend  on  appropriately  timed  (rare)  bursts  of  noise  and  a  simple 
diffusion  approximation  argument  is  inappropriate. 

In  order  to  relate  the  special  form  (3.1),  (3.2)  to  (2.5),  we  change 
coordinates  if  needed,  so  that  B  takes  the  form  of  the  column  vector 
(b,0,0,  •  •  •  ,0)  and  let  =  ''l  >  ^2  ~  0  =  b  =  o^. 

★ 

Computation  of  S^(T)  and  S*.  Let  us  put  the  variational  problem 
involved  in  (3.4)  into  a  more  enlightening  form,  and  work  with  the  special 
case  (3.1),  (3.2).  Let  6^  =  "Th®^ 

L(6,x)  =  s^up  [a^a^(x)a’Hx)  (gj-fjlx)) 

-  ^  a^o^(x)o^(x)aj] 

=  sup  [a|o'^(x) (6j-fj(x))  -  i  lojl^] 

=  lu(x)|^/2  , 


where  u(x)  =  0 j ^ (x)  (b^ -f ^ (x) ) .  Then 


(3.8) 


|u(<))(s)}  I  ‘  ds  , 


S^(T,<^) 


£ 

2 


0 


with 


(3.9)  i  =  f((())  +  o(4>)u  ,  4i(0)  =  X  . 

★  * 

Kith  the  form  (3.7),  (3.8),  the  calculations  of  S^(T)  and  are 

those  for  an  optimal  control  problem.  For  our  case  (3.6), 


Av  +  B  sin  (6-9) 

bu 

(3.10) 

f(x)  = 

Q 

X 

c 

II 

0 

_-Hv 

0_ 

Define  V(x)  =  S^.  Then,  purely  formally,  V(*)  satisfies  the 
Bellman  equation  [11,17], 

(3.11)  min[L”v(x)  +  u^/2]  =  0  ,  x  e  Gj 

V(x)  =  0  ,  X  c  , 

where  is  defined  by  L^g(x)  =  grad^g{x)  ♦  [f(x)  +a(x)u]. 

* 

KTiether  or  not  (3.10)  is  formal,  the  desired  solution  V(x)  =  can 

be  obtained  via  the  finite  difference  method  of  [15],  Chapter  9.9.6,  or  [16] 

where  a  method  for  a  slightly  simple  problem  is  discussed.  Let  h  denote 

a  finite  difference  interval  and  e.  the  unit  vector  in  the  ith  coordi- 

1 

nate  direction.  Define  the  h-Grid  G^  =  Gj n  {^k^e^h,  k.  =  all  integers). 

i 

By  an  appropriate  choice  of  finite  difference  scheme  for  the  derivatives 
in  (3.10),  one  can  obtain  a  finite  difference  solution  V^(x)  which  converges 
to  uniformly  in  any  compact  set  in  Gj.  The  conditions  required  in  [15] 

hold  for  our  case  (3.10),  and  G  =  Gj .  The  computational  method  also  yields  an 
approximation  u^(x),  x  €  Gj^,  to  the  optimal  control  u(x) . 


1* 
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In  the  computation,  we  used  the  set  =  {x;  |6-6|  <  tt}  in  order  not 
to  have  to  calculate  the  domain  of  attraction  of  x  =  0.  But  the  result  is 
the  same,  since  once  the  path  exits  G^,  it  will  leave  Gq  with  u(x)  =  0 
(no  additional  cost) . 


4 .  Exit  Paths  and  Measure  Transformations 

We  work  with  the  quantities  (3.5)  for  the  system  (3.6),  but  there 

are  identical  interpretations  for  the  left  sides  of  (3.7).  Let  there  be  a 

T  * 

finite  number  of  minimizing  functions  {ij>^(‘).  i^l^)  for  S^(T).  Then 
for  any  X  >  0  and  small  e,  the  overwhelming  proportion  of  the  paths 
which  exit  on  the  time  interval  (0,T]  starting  at  x,  exit  in 
u  N,  (4i(*)).  where  N  is  a  X-neighborhood .  But  the  probability  of  exit 

^  A  1  A 

on  [0,T]  is  small  for  small  e. 

Let  there  be  a  finite  number  of  minimizing  functions  ♦^(Ot  i  ^  for  S 
For  our  PLL  problem  these  occur  in  pairs  in  that  if  4(*)  is  a  minimizer, 
so  is  -^(O-  (Obviously,  there  are  an  infinite  number  of  minimizers 
^(•)>  since  we  can  always  let  ^^(•)  =0  on  any  interval  [O.tj],  t^  >0; 

but  we  do  not  need  to  count  these.) 

c 

For  small  e  ,  the  path  x  (•)  spends  most  time  in  a  small  neighborhood 

of  the  origin.  Occasionally  a  large  burst  of  noise  pulls  it  out  -  but,  with 

a  very  high  probability,  it  will  return  to  a  small  neighborhood  of  the  origin 

before  exiting  from  Gj.  Eventually,  however,  the  path  will  (w.p.l)  leave  Gj . 

When  it  does,  loosely  speaking,  it  leaves  in  a  small  neighborhood  of  the  path 

{^^(•)}-  ''e  can  quantify  this  as  follows.  Let  X  >  0  and  let  ^2  >  >  0 

be  small,  and  let  N,,  (0)  denote  a  p.  neighborhood  of  the  origin.  Then  [6] 

^  .  1 
I 

for  X  €  N  (0) , 

(4.1)  lira  P^{x^(t)  €  IJ  N^(^^(t))  until  the  exit  time  j 

i 

X  (•)  exiti"  Gj  and  does  not  return  to  (0)  after  leaving 

N.  (0)}  =  1 
^2 

There  is  only  one  pair  of  optimal  paths.  The  parts  of  the  functions 


' ;  >r.:  -r  <_  V  ►  -  -  -r  „  ir>  W-^  «r^  ' 


{((:|^(’).  on  any  finite  interval  [O.T^]  converge  to 

on  [0,T^]  as 

It  is  clear  from  this  discussion  that  the  most  likely  noise  sequences 
which  lead  to  exit  are  of  the  form  (write  <)!(*)  =  (<l)j  (•)  form  (3.1)) 


(4.2)  w(t)  'v- 


7  o'^(4(t))[«^(t)  -  1^(0)  -  I  fjKs)]ds]  . 


where  !(•)  is  a  minimizer  for  Sq.  From  (4.2),  we  can  get  a  clear  idea 
of  when  the  noise  (leading  to  exit)  will  be  large,  and  in  which  direction 
it  will  push  the  system. 


A  likelihood  function  interpretation  of  S^(T)  and  S^.  It  is 

suggestive  to  view  (3.4)  as  selecting  an  exit  path  which  maximizes 

£; 

a  likelihood  function.  Since  for  each  c  >  0,  x  (•)  is  not  differentiable, 
one  cannot  speak  of  a  likelihood  function  in  a  strict  sense.  But,  for  the 
Gaussian  case  (3.1),  one  can  get  an  intuitively  reasonable  maximum  likeli¬ 
hood  interpretation.  Even  from  this  the  estimates  (3.5)  do  not  follow 
readily  without  the  large  deviations  formalism. 

Consider  the  discrete  parameter  process 


(4.3)  +  Af(Xj’“)  +  ^ea(xj’“)c.  . 


where  are  i.i.d.  and  normal  (0,1),  The  likelihood  function  of 

{X^’^,  i  <  T/A)  evaluated  on  the  path  =  4(iA),  iA  £  T,  is  (4.4)  times 

factor  not  depending  on  the  path. 

.  T/A-1  2 

(4.4)  L  (T,4)  =  JI  exp - lu(iA)l 

0  2e2 

where 


(((iA  +  A)  =  (j;(iA)  +  Af(4(iA)}  +  Ao  (4  ( iA) )  u  ( i  A) 


We  have 


(4.5)  S(T,(^)  =  -lim  e2  log  L^(T,(^)  . 

A 

Let  denote  the  piecewise  linear  interpolations  of 

with  interval  A,  and  the  escape  time  from  G  of  x^’^(*)-  By  the 

standard  methods  of  the  theory  of  large  deviations  [9,10],  it  follows  that 
if  A  -»•  0  as  e  -+•  0  and  under  broad  conditions  on  G 

lim  c2  log  p  {x^’^(t)4G,  some  t  ^  T]  =  -S*(T) 
e,A  ^  ^ 

(4.6) 

lim  e2  log  ~  "^0  ’  x  c  G^  . 

e ,  A 

If  (4.6)  does  not  hold  for  a  given  G,  then  it  will  hold  for  a  small 
perturbation  of  G.  Also,  (4.6)  holds  for  the  discrete  time  form  of  our 
special  case  (3.6) . 

These  estimates  provide  justification  for  simulating  the  discrete 
parameter  system  in  lieu  of  x^(*)' 

Importance  sampling.  Now  that  we  know  what  the  'most  likely  exit 

paths' are,  we  are  ready  to  define  the  measure  transformation  used  to 

facilitate  the  simulations.  Following  the  idea  in  [14],  we  transform  the 

measure  so  that  exit  on  [0,T]  is  not  a  rare  event.  T  will  be  large 
★  * 

enough  so  that  "v  Sj^(T).  To  get  the  measure  transformation,  we  use 
the  approximation  to  the  optimal  controls  u^(*)  given  by  the  computa¬ 
tional  procedure  discussed  at  the  end  of  Section  3.  For  x  G  G^  =  G  but  not 
in  the  grid  G^^,  we  define  u^(*)  by  a  linear  interpolation  over  the  values 

V, 

in  adjacent  grid  points.  For  simplicity,  we  write  u(x)  below  for  u‘  (x) . 

Write  the  system  in  the  form  (4.7)  with  associated  measure  Pq.  Ke  work 
on  an  arbitrary  interval  [0,T],  Define  x^(*)  on  [0,T]  by  (4.8),  with 
associated  measure  P^. 
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(4.7)  dx^  =  f(x^)dt  +  Eodw 

(4.8)  dx^  =  f(x^)dt  +  ou(x^)dt  +  eodw  . 

Again,  using  the  notation  x  =  (x^.x^),  o  =  (Oj,0),  •••,  the  Radon- 
Nikodyn  derivative  is  [18]: 

exp  —  [i  [  lu(x(s)) j^ds 

£2  Jq 

-  I  u’ (x(s))o'^ [dXj(s)  -  (x(s))ds]j 


N 


I'e  use  a  discrete  form  in  the  simulations.  Thus,  define  the  processes 

(with  associated  measures  P^’^,  P^’^,  respectfully,  on  [0,T/e]) 

M.lOa)  =  X^’^  +  Af(X^’^)  +  c/Ka^. 

1+1  1  1  1 

(4.10b)  X^^j  =  +  Af(X^’^)  +  AO  u()^’^)  +ev^oC.  . 

Ke  always  let  u(^’^)  =0  after  the  first  exit  time  from  G  =  G^.  Ke 
have,  for  =  x!’^  ,  -  -  Af,  ()^’^)  , 

’^i  1,1+1  1,1  1  1 

dP^’^  T/A-1  ,  n  *  A  1  aI 

(4.11)  (w)  =  n  exp  —  ^  lu(x]’^|^  A-u'(X^'^o'^i|;^’'M  . 

dpE.^  0  e2  Le  1  1  i  1  J 


1^:=- 


Ke  will  simulate  using  (4.10b).  Let  M  denote  the  number  of  (mutually 
independent)  runs,  indexed  by  ,  i  5 M.  Define  the  set  A  whose 
probability  is  of  interest; 

A  =  {paths  exiting  G  =  Gj  by  time  step  T/A,  starting  at  x(0)  w  0}  . 
Then  (4.12)  is  an  unbiased  estimate  of  P^’^{A),  the  escape  probability 
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5.  Simulations  with  e(t)  =  6^  =  0 

We  use  a  first  order  filter,  and  the  simulated  PLL  equations  are  the 
discrete  time  form  of 

dx^  =  -ax^dt  +  b(sin  x^dt  +  edcc)  , 

(5.1) 

dx^  =  -Xjdt  ,  x^  =  (e-e^) 

In  all  cases,  a  =  2,  and  the  results  in  Tables  1  to  3  are  tabulated 
according  to  the  value  of  b.  In  order  to  best  compare  the  simulation  and 
the  theoretical  estimates,  we  write  (4.12)  in  the  form  exp  -S^(T)/e2  and 
tabulate  ^ (T) .  We  use  (T)  for  the  equivalent  sample  probability  of 
escape  on  [0,T] .  There  is  clearly  a  very  close  agreement  between  the 
theoretical  predictions  and  the  results  of  the  simulations. 

The  (#  escape)  denotes  the  number  of  simulated  paths  escaping  on  [0,T] 
under  the  transformed  measure  (for  the  importance  sampling) ,  not  the  measure  of 

(5.1) .  It  indicates  the  savings  in  the  cost  of  simulation.  The  'eigenvalues' 
denotes  the  eigenvalues  of  the  noiseless  (5.1),  linearized  at  x  =  0. 
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Table  1 


b  =  1,  eigenvalues  (1,1),  A  =  .02,  200  Runs 


T 

e 

#  Escape 

* 

% 

S^(T) 

r 

(T) 

io-2« 

i 

3.5 

.2 

15 

2.55 

2.59 

.8  X 

f 

.4 

85 

2.55 

.95  X 

10"^ 

.6 

121 

2.55 

.84  X 

10-^ 

.8 

137 

2.39 

.  24  X 

10"^ 

r 

5 

.2 

42 

2.55 

2.56 

.  16  X 

10-2^ 

•'  ■- 

.4 

155 

2.46 

.21  X 

10-6 

-  • 

. 

.6 

171 

2.39 

.13  X 

10  “ 

.8 

174 

2.06 

.40  X 

10'^ 

■ ' 

*> 

.1 

184 

1.80 

.17 

:.v 

[L, 

7.S 

.2 

129 

2.55 

2.5 

.81  X 

10-2^ 

v\-- 

.4 

192 

2.39 

.33  X 

10-6 

.6 

194 

2.22 

.21  X 

10"^ 

m 

.8 

196 

1.99 

.45  X 

10-' 

Table  2 


2,  eigenvalues  (-1±»^),  A  =  -02,  200  Runs 


Escape 

"o 

S^(T) 

P^(T) 

5 

1.28 

1.36 

o 

X 

00 

58 

1 .41 

o 

X 

95 

1.32 

.25  X  10' 

109 

1.37 

,12 

40 

1.28 

1.29 

.12  X  10 

128 

1.25 

X 

o 

150 

1.09 

.49  X  10 

157 

.84 

,23 

107 

1.28 

1.25 

,29  X  10 

179 

1.12 

.93  X  10 

1  7Q 

1.02 

X 

o 

Table  3 


Comment  on  the  Choice  of  T 


To  get  associated  measure  transformation,  we  need 

controls  which  are  time  dependent  (hence  more  computation  and  memory  is 
needed).  In  applications,  the  value  of  T  is  not  very  important  (multi¬ 
plying  or  dividing  a  reasonable  T  by  a  constant  k  also  yields  useful 

results).  The  c-effects  are  much  more  pronounced.  For  'large'  T, 

★  ★ 

S  (T)  -  S  and  most  of  the  'control  activity'  for  the  optimal  escape 

A  u 

path  occurs  long  before  T.  In  our  problem  the  optimal  exit  paths  exit  at 
6-6  =  ±7T  ,  V  =  0,  and  take  an  infinite  time  to  reach  these  points. 

Because  of  this,  we  chose  T  as  follows.  Using  the  optimal  control  u(x) 

and  the  noiseless  system,  we  start  the  trajectory  very  close  to  the  origin 

* 

(not  at  X  =  0,  since  u(x)  =  0)  and  let  T  denote  the  time  the  path  takes 

★ 

to  get  to  a  position  x  near  the  exit  point  where  is  very  small.  Our 

* 

larger  T-values  are  about  1.25  T  . 

niscus.sion  of  the  Data 

As  can  be  seen  from  the  data,  even  for  the  smaller  T,  the  measure 

★ 

transformation  obtained  with  the  (nearly)  optimal  u(*)  for  yields 

excellent  results.  If  the  prediction  is  close  to  the  sample  estimate  for 

large  T  and  small  e  (say  e  =  .2),  then  we  are  very  likely  justified  in  using 

-  e  * 

the  small  e  values  of  S  (T)  as  estimates  of  Sq(T) ,  and  then  we  can  compare 
these  to  the  sample  estimates  for  large  values  of  e. 

The  role  of  the  measure  transformation  is  made  very  clear  by  the 
tables.  E.g.,  for  c  =  0.2,  b  =  1  and  T  =  3.5,  the  probability  of  escape 
is  :  10  ‘  .  An  estimate  of  such  a  quantity  would  require  an  impossible 
amount  of  computer  effort.  But,  via  the  importance  sampling  measure  trans- 


formation,  in  200  runs  there  were  15  escapes.  For  the  transformed  measure 
this  corresponds  to  a  (not  small)  probability  of  0.075.  Under  the  inverse 
transformation,  it  becomes  the  true  estimate  of  the  order  of  lO'^®.  The 
method  is  efficient,  indeed. 

The  size  of  the  noise  effects  can  be  judged  by  the  estimated  escape 

probability.  For  realistic  systems,  one  would  expect  small  errors 

-4 

(perhaps  of  the  order  of  10  or  smaller).  Yet  our  predicted  results  are 
quite  good  even  for  values  of  e  which  correspond  to  much  larger  escape 
probabilities . 

As  e  increases,  the  theoretical  predictions  increasingly  understate 
the  escape  probability;  the  noise  is  stronger  and  it  is  unlikely  that  the 
random  escape  paths  of  x^(0  are  concentrated  in  a  small  tube  about  the 
optimal  escape  paths. 

The  vector  field  of  the  noiseless  system  (b  =  1.0)  is  in  Figure  5.1, 
and  for  the  optimally  controlled  system  in  Figure  5.2.  The  control 
activity  is  negligible  near  the  origin  or  near  x^  =  0,  X2  =  ±tt,  or  for 
large  values  of  x^.  The  most  likely  noise  bursts  which  lead  to  exit  occur 
when  the  filter  state  Xj  is  not  large.  For  small  e,  the  exiting  paths 
were  usually  in  a  small  neighborhood  of  the  optimal  paths  (0  y)  ,  or 
(0  -►  -y) ,  as  indicated  in  Figure  5.2.  Even  when  e  is  rather  large,  the  most 
likely  exit  paths  are  reasonably  close  to  the  optimal  paths.  Two  typical  cases 
are  plotted  in  Figure  5.3,  one  exiting  at  (0,11)  and  the  other  at  (0,-it). 


NOISELESS  PATHS,  b  =  1.0 
FIGURE  5.1 
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6 .  Time  Varying  Phase:  The  Model 


In  order  to  illustrate  the  interactions  of  signals  and  noise  which 
lead  to  the  most  likely  escape  routes,  and  to  check  the  range  of  validity 
of  the  large  deviations  estimates  when  the  signal  varies  with  time,  the 
following  version  of  a  pulse  phase  modulation  system  was  considered.  The 
data  signal  is  a  sequence  of  pulses,  each  of  value  ±1  and  constant  over  an 
interval  of  unit  length.  When  the  sequence  changes  from  a  +1  to  a  -1,  the 
transmitted  signal  phase  ©(tj  goes  from  +it/4  to  -it/4,  and  conversely 
when  the  data  sequence  jumps  from  -1  to  +1.  The  probability  of  changing 
sign  is  pc  (0,1).  The  object  of  the  PLL  is  to  track  6(t)  (hence,  to  track 
the  data  sequence) .  Often  the  transmitted  signal  for  such  a  data  sequence 
changes  from  ±tt/2  to  +tt/2,  and  a  form  of  the  PLL  which  is  better  suited 
to  digital  data  is  used.  But  the  system  used  here  well  illustrates  the 
general  principles. 

The  systeiTi  is  (for  a  general  filter) 


(6.1)  dv'^  =  Av^dt  +  B 


cdw  +  sin(e-0*^)(Jt 


I 


d(e-e^)  =  -Hv'^dt  +  dJ  , 


where  J(")  is  a  jump  process  which  jumps  by  ±ir/2  depending  on  the  direction 


of  change  of  the  sign  of  the  binary  data  sequence.  We  take  the  paths  of  ©(•) 

^  c 

and  6  (•)  to  be  right  continuous. 

The  comments  on  large  deviations  in  Sections  3  and  4  all  hold  here,  by 
proofs  which  are  very  close  to  those  used  to  support  the  assertions  in  those 
sections.  Owing  to  the  presence  of  the  time  varying  signal,  the  action  functions 
must  depend  on  time  as  well  as  on  the  direction  of  the  next  possible  change  in 
J(*)*  The  appropriate  action  functional  is 


(6.2) 


(T,<^,J) 

T  ,  A 


T 
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where 


(6.3)  d(f  =  f(4i)dt  +  Budt  + 

♦(t)  =  X 


Here  x  is  the  initial  time,  and  (|:(t)  =  x  the  inital  condition  and  J  the 
future  jump  path.  J(’)  also  acts  as  a  control  force  here  -  but  is  confined  to 
be  a  possible  signal  path.  Define 


(6.4)  S*  (T,±)  =  inf  S  (T.(t>,J) 


(6.5)  S*  ^(±)  =  lim  S  (T,±)  . 

^  ^  T-*®  ' 

where  the  inf  is  over  all  <()(•)  and  J(*)  satisfying  (6.3)  with 
(t)(t)  i  Gj  for  some  t  $  T,  and  the  next  possible  jump  of  J(*)  after  t 
is  ±.  Then,  via  mild  modifications  in  the  standard  methods 


(6.6) 


lim  log  P  {x^ (t)  i  G, ,sorae  t ^ t ? T)  =  -S  I 

X  j  T  A  X  j  T 


Also  the  analog  of  (3.7a)  holds. 

* 

The  limit  S  (±)  =  V(t,x,+)  has  an  optimal  control  interpretation 

V(t,x,+)  =  min  (S  (T,(|),J);  4)(0)  =  x,  (6.3)  holds, 

T,(f,J 

iJiCT)  ^  Gj  ,  next  possible  jump  after  t  is  ±, 
V(t,x,±)  is  periodic  in  t;  V(n,x,±)  =  V(0,x,±),  all  n,  and  there  is  a  possible 
discontinuity  at  t  =  integer  value.  Because  of  this,  V(t,x,±)  satisfies  the 
Bellman  equation  (6.7),  (6.8),  for  x  €  Gj  with  V(t,x,±)  =  0  for  x  ^  G. . 


(6.7)  0  =  min  +  L^V(t,x,±)  +  u^/2]  for  0  $  t  <  1  , 

U  V  T 


(6.8) 


V(1  ,x,±)  =  min  {V(0,x±  (^ ^  )  ,+ )  ,  V(0,x,±)} 


The  numerical  procedure  alluded  to  in  Section  3  can  still  be  used-- 


We  use  the  state  variable  triple  (t.x.i'.  with  x  £  and  t  €  (0,T] . 


All  the  comments  in  Section  4  concerniic  the  most  likelv  exit  paths 
continue  to  hold,  except,  of  course  the  exit  paths  and  solutions  to  the 
variational  problems  might  have  jumps  at  integer  values  of  time. 

The  importance  sampling  method  and  measure  transformation  are  also 


almost  identical  to  those  described  in  Section  4.  Ke  work  with  the  discrete 


£  ^  p  A. 

parameter  systems  on  (0,T]  (with  measures  P  ’  and  P  ’  ,  respectively. 


0  u 

where  u  is  the  optimal  policy  obtained  from  (6.7),  (6.8)). 


(6.9)  a  Af()^’^  a  u  +  e/A 


^i  ^ 


"  1 
J(iA+A) -J (iA) j 


(6.10)  =  xj’^  +  Af(xJ’^)  4  e/a  M  c. 


0 


J(iA+A) -J(iA) 


Then  with  u(x,t,j)  denoting  the  optimal  control  at  time  t,  when  the 
next  possible  jump  can  be  j ,  we  use  the  measure  transformation 


dP 


e ,  A 


(6.11) 


0 


T/A-1 


1 


(w)  =  H  exp  ^ 
dP  ’  0 


J  |u[x^’^,iA,j(iA))  I'^A 


-u'  (Xj  ’^,iA,  j  (iA))o"^iJi^ 


where  o  =  b  and  ,  -  X^’^  -  Af,()u’^) 

1  I  1 ,1  +  1  1,1  1^1 


T  was  selected  as  in  Section  5  for  the  constant  phase  system. 


Simulations  Kith  Pulsed  Phase 


The  simulations  all  started  with  x(0)  at  zero,  and  with  the  next 
possible  jump  in  phase  being  positive.  Two  types  of  jump  sequences  J(0 
were  used.  In  the  first,  denoted  by  'random  jumps,'  the  phase  changed  by 

±7r/2  at  each  unit  of  time  with  probability  p  =  1/2.  Solving  the  varia- 

*  -  _ 

tional  problem  for  ^(  +  )  yields  a  unique  minimizing  pair  (<j>( *)  ,J( •) )  >  where 

^C’)  is  an  'optimal  escape  path',  starting  at  x  =  0  at  time  t  =  0.  The  j(*) 

might  have  one  or  many  jumps  in  it.  When  J(*)  was  used  in  lieu  of  random 

jumps,  the  results  are  labeled  'optimal  jumps'  in  the  tables  below. 

Tables  4  to  6  list  the  results  of  the  simulations  (x(0)  =  0,  tCO)  =  o) , 

and  excellent  agreement  between  the  sample  estimates  and  the  theoretical 

predictions.  For  example,  in  Table  5,  where  e  =  .6  and  the  estimated 

escape  probability  is  the  very  high  0.37,  there  is  still  good  agreement. 

The  results  of  the  simulations  with  the  'optimal  jumps'  should  be  somewhat 

★ 

more  consistent  with  the  theoretical  values  of  Sq  qC±)  than  would  the  'random 

* 

jump'  simulations,  since  it  is  the  optimal  jumps  which  are  used  to  compute 
But  agreement  is  good  for  both  cases.  The  sample  S^(T)  values  for  the  random  jump 
case  are  larger  than  for  the  optimal  jump  case,  as  expected,  since  the  optimal 
jumps  are  calculated  to  help  force  the  system  out  of  G^.  But,  as  c  0,  the  two 
estimates  are  close.  For  small  e,  the  exit  paths  were  in  a  small  neighborhood  of 
the  optimal  exit  path.  Two  optimal  exit  paths  are  plotted  in  Fig.  7.1,  for  b  =  1 
and  b  =  0.75,  when  the  next  possible  jump  is  +it/2  in  the  first  case,  and  -rjl 
in  the  second  case. 

For  the  case  b  =  1,  the  optimal  path  waits  until  time  x  =  1,  with  zero  control 
u(x)  then  jumps  to  (0,7r/2),  then  a  large  control  effort  moves  it  close  to  the 


optimal  exit  point  (0,m),  during  which  time  there  are  no  further  jumps.  For  the 


case  b  =  0.75,  the  optimal  exit  path  involves  waiting  at  the  origin  with 
zero  control  until  a  time  <  1,  then  being  controlled  to  in  [T2,l], 

then  jumping  to  y,  at  x  =  1  and  finally  having  a  burst  of  control  effort 
which  moves  it  close  to  the  optimal  exit  point  y^,  during  which  there  are  no 
further  jumps.  The  intervals  of  heav>'  control  effort  (and  the  corresponding 
directions)  correspond  to  the  most  likely  intervals  during  which  a  burst  of 
noise  (and  its  corresponding  direction)  will  lead  to  exit.  The  first  burst  of 
noise  on  [T2,l]  positions  the  path  to  take  best  advantage  of  the  next  jump.  Of 
course,  even  if  the  first  burst  of  noise  occurs  ’on  schedule',  the  next  jump 
might  not  occur  'on  schedule'.  Then,  most  likely,  the  system  will  drift  back 
to  near  the  origin.  What  we  have  described  are  the  paths  in  small  neighborhoods 
of  which  exit  is  most  likely  to  occur.  The  sample  paths  some  time  before  the 
'final  sequence'  leading  to  exit  might  be  rather  complex- -owing  to  the  jumps. 


Table  4 


Eigenvalue  (-1,-1),  A  =  0.025,  b  =  1 
T  =10  Sec,  500  Runs 


Optimal  Jumps: 


T  e 

**  Escape 

^0,0^^^ 

S^(T) 

P^(T) 

10  .2 

24 

.79 

.82 

.13  X  10' 

.3 

145 

.82 

.11  X  10' 

.4 

251 

.79 

.72  X  10' 

.5 

335 

.71 

.58  X  10' 

.6 

399 

.63 

.17 

Random  Jumps,  p  = 

1/2 

10  ,2 

25 

.79 

.83 

.11  X  10' 

.3 

125 

.87 

.66  X  10 

.4 

219 

.86 

.48  X  10 

.5 

265 

.86 

.32  X  10 

.6 

312 

.84 

.97  X  10' 

15  .2 

64 

.79 

.81 

.17  X  10 

.4 

274 

.82 

.59  X  10 

5  .2 

10 

.79 

.84 

.82  X  10 

.4 


95 


.98 


.22  X  10 
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Table  5 

(2,2)  Eigenvalues  (-ll/T),  A  =  0.025,  b  =  2 
T  =  6  Sec,  500  Runs 


Random  Jumps,  p  = 

1/2 

T  E 

#  Escape 

S^(T) 

^(T) 

6  .2 

86 

.40 

.40 

o 

X 

.3 

187 

.41 

.11  X  10 

.4 

233 

.44 

.63  X  10 

.5 

299 

.41 

.20 

.6 

367 

.36 

.37 

.7 

407 

.31 

.53 

8  .3 

228 

.40 

.38 

.14  X  10 

Table  6 


Eigenvalues  (-1/2, -3/2),  A  =  0.025,  b  =  0.75 
T  =10  Sec,  500  Runs 


Random  Jumps,  p  =  1/2 


e 

^  Escape 

^(T) 

P^(T) 

.2 

180 

1.15 

1.11 

o 

X 

00 

.3 

235 

1.16 

.26  X  10 

.4 

279 

1.24 

.42  X  10 

.5 

315 

1.18 

00 

00 

X 

o 

.6 

319 

1.23 

X 

o 

.7 

354 

1.19 

o 

X 

00 

00 

.8 

373 

1.14 

.17 

.9 

409 

1.08 

.26 

.2 

124 

1.15 

1.11 

X 

1— » 
o 

.3 

183 

1.20 

c 

X 

.4 

242 

1.24 

o 

X 

.5 

239 

1.28 

.61  X  10 

Figure  7.1  indicates  an  optimal  exit  path,  but  it  does  provide  a  picture 
of  where  and  when  the  controls  are  strongest  (equivalently,  where  and  when 
the  noise  bursts  which  lead  to  exit  are  likely  to  occur).  We  will  now  describe 
part  of  the  time  dependent  control  behavior,  for  the  case  b  =  1  on  the  interval 
[0,1),  when  the  next  possible  jump  in  phase  (e-8)  must  be  negative.  Since 
the  control  is  added  to  the  uncontrolled  dynamics,  it  will  be  useful  to  refer 
to  Fig.  5.1. 

There  are  2  regions  where  the  effects  of  the  control  are  large  and  its 
effects  obvious.  In  a  small  neighborhood  below  and  to  the  right  of  (O.r) 

(in  the  phase  plane)  the  controls  are  large  and  positive  and  essentially 
independent  of  time.  Here,  the  control  is  trying  to  complete  the  job  of 
pushing  the  path  out  of  G^.  The  relatively  small  time  dependence  is 
probably  due  to  the  fact  that  for  the  deterministic  optimal  control  problem, 
there  need  not  be  another  jump.  If  the  process  is  close  to  (0,r),  it  is 
'cheaper'  to  treat  it  as  though  there  were  no  additional  jumps.  The  second 
area  where  the  controls  are  large  is  in  the  brood  region  below  the  origin,  but 
here  the  behavior  is  quite  time  dependent.  The  controls  act  to  counteract  the 
unforced  system's  damping,  so  as  to  position  the  process  such  that  a  jump  of 
-7t/2  will  (or  nearly  will)  drive  the  path  out  of  Gj .  Directly  below  the 
origin  the  control  takes  negative  values,  which  increase  as  x  -►  1 .  The  dominant 
effect  here  is  to  'slow  down'  the  path  and  prevent  it  from  being  pushed  into 
the  right  half  plane,  where  it  would  be  swept  upward  and  further  away  from  the 
-t/2  phase  level  before  the  next  'negative'  jump  occurs. 
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